Entropy measures for networks: Toward an information theory of complex topologies 
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The quantification of the complexity of networks is, today, a fundamental problem in the physics 
of complex systems. A possible roadmap to solve the problem is via extending key concepts of 
information theory to networks. In this paper we propose how to define the Shannon entropy 
of a network ensemble and how it relates to the Gibbs and von Neumann entropies of network 
ensembles. The quantities we introduce here will play a crucial role for the formulation of null 
models of networks through maximum-entropy arguments and will contribute to inference problems 
emerging in the field of complex networks. 
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Complex networks 0, 0, 0, 0] are found to character- 
ize the underlying structure of many biological, social 
and technological systems. Following ten years of active 
research in the field of complex networks, the state of 
the art includes, a deep understanding of their evolution 
[H, an unveiling of the rich interplay between network 
topology and dynamics Q and a description of networks 
through structural characteristics Nevertheless, we 

still lack the means to quantify, how complex is a com- 
plex network. In order to answer this question we need 
a new theory of information of complex networks. This 
new theory will contribute to solving many challenging 
inference problems in the field 0, 0, @] . By providing an 
evaluation of the information encoded in complex net- 
works, this will resolve one of the outstanding problems 
in the statistical mechanics of complex systems. 

In information theory 0] entropy measures play a key 
role. In fact, it is well known that the Shannon en- 
tropy and the von Neumann entropy are related to the 
information present in classical and quantum systems, re- 
spectively. Moreover, the afore mentioned measures also 
have statistical mechanics interpretations. Traditionally, 
in statistical mechanics, for configurations drawn from 
canonical ensembles, the Shannon entropy corresponds to 
the entropy for classical systems, while the von Neumann 
entropy provides the statistical description of quantum 
systems. 

In the context of complex networks a number of dif- 
ferent entropy measures have been introduced d, 0, 
ToL 11, Tl, Hi- I n R- e f- 01 the Gibbs entropy per node, 
in a network of N nodes, denoted E, was introduced for 
microcanonical network ensembles following a statistical 
mechanics paradigm. Microcanonical network ensembles 
arc defined as those networks that satisfy a given set of 
constraints. Examples of some popular constraints in- 
clude, fixed number of links per node, given degree se- 
quence and community structure. The Gibbs entropy of 
these ensembles is given by 



£ = -logAf, 



(1) 



where N indicates the cardinality of the ensemble, i.e., 
the total number of networks in the ensemble. As demon- 
strated further in [§] the statistical mechanics formal- 
ism enables us to develop canonical network ensembles 
where the structural constraints under consideration are 
satisfied, on average. In classical statistical mechanics 
the microcanonical ensemble is formed by configurations 
having constant energy E, while the canonical ensemble 
is formed by configurations having constant average en- 
ergy (E). By analogy, in the theory of random graphs 
the G(N, L) graph ensemble is formed by networks of N 
nodes with a constant total number of links L. In the 
conjugated-canonical G(N,p) ensemble, however, the to- 
tal number of links is Poisson distributed with average 
(L) = p(N — 1). This construction of microcanonical 
and conjugate-canonical ensemble can be further gener- 
alized [9( to network ensembles with more elaborate sets 
of constraints. For example we can define microcanoni- 
cal network ensembles with given degree sequence 
and canonical network ensembles (based on hidden vari- 
ables [3, EH) in which each node i has fc, links, which is 
Poisson distributed with average (fcj) = 

In this letter we show for this new statistical mechanics 
framework of networks, first, that the entropy of canoni- 
cal network ensembles is related to the Shannon Entropy 
and second, that canonical network ensembles satisfy a 
principle of maximal Shannon entropy. Moreover we will 
study to what extent canonical and microcanonical net- 
work ensembles are equivalent. Finally we will discuss 
the relation between the Shannon entropy of a canoni- 
cal network ensemble, S, and the recent definition of von 
Neumann entropy of networks, Svn, recently introduced 
in Ref. [IH of interest in the field of quantum gravity [IH . 

Gibbs entropy of a microcanonical network ensemble. 
Microcanonical network ensemble arc formed by network 
satisfying a given number of constraints. Following the 
lines of reasoning provided in [§], on specifying the full 
set of constraints and number of nodes N in the net- 
works, one may introduce a partition function Z for the 
ensemble. This partition function counts the number of 
networks, defined by their adjacency matrices {a^ }, that 
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simultaneously satisfy all the constraints under consid- 
eration. The adjacency matrix describes an undirected 
network, i.e., ciij = a^, where each element takes some 



positive integer values, a,j £ a, where a C 



that 



dicates the weight of a link between nodes i and j. For 
simple (connectivity) networks we take £ {0, 1} while 
for weighted networks «y 6 N. Thus, we have 

Z = J2 n 5 ( constraint fe({ a y})) e ~ i:,<jEa/llj(a),5aiJ ' Q > 
{»«} fc 

_(2) 

where the fields hij (a) play the usual role of auxiliary 
fields in statistical mechanics. Finally the Gibbs entropy 
S, defined by Eq. ([1]), and the probability 7Ty(o;) of hav- 
ing a link between nodes i and j, with weight a, arc given 
by 



NY; = 



7Ty(a) 



l°8^1/,y(a)=0 V(t,j» 

aiog-z 

dh i:j (a) ' 



(3) 



Entropy of a canonical network ensembles. The canon- 
ical network ensemble can be built starting from the 
marginal distribution 71"^ (a), given by Eq. ((3]). For a net- 
work of TV nodes, for each pair of nodes, one draws 
a link of weight a with probability 7Ty (a). The proba- 
bility II of the canonical undirected network ensemble, 
defined by its adjacency matrix {a^}, is therefore given 
by 



(4) 



i<3 



for which the log-likclihood function is given by 

£ = -^log7r ij (ay). (5) 



%<3 



The entropy of a canonical ensemble is the logarithm of 
the number of typical networks in the ensembles and is 
given by 



S = (£)n = - y y T^ij («) log 7r 4i (a) , 



(0) 



i<j a 



which takes exactly the form of a Shannon entropy. We 
will therefore call this quantity the Shannon entropy of a 
network ensemble. In particular, for the case of a simple 
undirected network, where a £ {0, 1}, we have 

S = ~^Pij logpij - ^(1 - pij) log(l - p^) , (7) 



i<3 



where p^ = 7ry (l) is the probability of having a link 
between nodes i and j. 

Maximizing the Shannon entropy of the network 
subjected to different types of constraints gives rise 
to maximum-entropy ensembles and generalizing the 



maximum-likelihood arguments of [161 ]. In the following 
we will consider few examples, of such constraints for the 
cases of simple undirected networks. 

Fixing the total number of expected links, p^ = 
L, the maximum-entropy ensemble is G(N, {pij}), with 
Pij = p = L/(N(N— l)/2). Alternatively, if we constrain 
the expected degree of each node i, i.e., Ki = J2jPij> the 
probabilities in the maximum-entropy ensemble take the 
form p^ = 6i9j/(l + QiOj) where 9i are hidden- variables 
fixed by the constraints. This ensemble is the canonical 
conjugated to the microcanonical ensemble of networks 
with fixed degree sequence {k^}. In table I we general- 
ize this construction and report the form of maximum- 
entropy network ensembles satisfying a different sets of 
constraints. We leave to the reader the construction of 
maximum-entropy weighted network ensembles related 
to the canonical ensembles discussed in Refs. 0, [l7| . 
The marginal probability for the microcanonical and con- 
jugated canonical ensembles are equal by definition, but 
in order to prove the equivalence between the two ensem- 
bles also the entropy per node S and S/N must be equal 
in the thermodynamic limit. 

Comparison between the entropies of the G(N, L) and 
the G(N,p) ensembles. We study first the relation be- 
tween the Gibbs entropy S and the Shannon entropy per 
node for random graphs, defined for the G(N,L) and 
G(N,p) ensembles, respectively. The Gibbs entropy in 
the G(N, L) ensemble is given by Q 



N(N-l) 

7V£ = log I 2 



(8) 



As mentioned earlier, the corresponding probability of 
each link in the conjugate G(N,p) ensemble is given by 
Pij = p = 2L/(N(N — 1)). Inserting this probability in 
the definition of the Shannon entropy, Eq. ([7|), we get 



S ' N+ 2N 



log 



/N(N 



-r 



V 2L 



Therefore the Gibbs entropy S and the Shannon entropy 
per node S/N of random graphs are equal in the ther- 
modynamic limit N — * oo. 

Comparisons between the network ensembles with given 
degree sequence and structural cutoff. The microcanoni- 
cal ensemble of networks with given degree sequence {fti} 
has been fully characterized in [9( . For simplicity, we con- 
sider the Gibbs entropy per node S in the case where the 
maximal connectivity of the nodes satisfy a structural 
cutoff, i.e., k max < y/ (k)N. In this limit the statistical 
mechanics treatment gives the Bender formula [l8| and 
the Gibbs entropy per node E is given by 



iVS = log[(2L-l)!!]-^log(/ ti !)- 7 



1 (Hi 



(9) 



In the conjugate-canonical ensemble, the probability of 
having a link is given by ptj = j^y^- Inserting this ex- 
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Ensembles 


Probabilities Pij/(l — Pij) 


Conditions 


Given expected 
number of links L 


p/(l-p) 


pN(N -1)/2 = L 




Given expected 
community structure{yl (3 <q i } 


w( qi , qj ) 




^(Sl <?') \ q ^ q l =Z~2ijPijfiqi,q<5qj,q' 


Given expected 
degree sequence {ni} 


OiOj 


K i = Pij 




Given expected 
degree sequence {fti} 
community structure {A(q,q')} 




K>i = 2~2j Pij 


A (q,q')\ q9 iq> =2ZijPij S H,1 S qj,q' 


Given expected 
degree sequence {ni} 
and number of link at distance d, B(d^) 




Hi = },j Pij 


B ( d ) = EijPy^.d 


Given expected 
degree sequence {m} 
and number of triangles 
for each node {Ti} 




K i = 22 j Pij 
T i = HjkPijPjkPki 


/y ~ X]fc PikPkj 

9ij = EfcP»fc Q fePfcj 



TABLE I: Maximum-entropy networks ensembles with given set of constraints. The community of each node is associated with 
a Potts variable qi. The distance of the nodes is binned and indicated by a discrete variable dij = d. The hidden variables of 
each ensembles {8i}, W(q,q'), W(d), {ofi}, {fij,9ij}, are fixed by respective conditions specified in the table. 



pression into Eq. ([7]) we get for the Shannon entropy of 
the ensemble 

E = ^-E[ 1 °g(^ ! /«'e- K -))] + O(l0g(A r )/iV) (10) 



We observe that the entropy per node E and the Shan- 
non entropy per node S/N of the canonical conjugated 
network ensemble are not equal in the thermodynamic 
limit. This implies, for example, that the entropy per 
node of regular networks is smaller than that of a Pois- 
son network with same average degree. In particular, 
suppose we take, for regular networks, Ki = c and for the 
conjugated-canonical Poisson network, fcj to be a Poisson 
distributed random variable with a mean (ki) = K, = c. 
The entropy of regular networks E^ and the entropy of 
Poisson networks Ser are related by the expression 



E fi = Ser/N - log ■ 



Ser/N -±log(c), (11) 

where in the last expression we have taken the Stirling 
approximation valid for large c. 

The non-equivalence of E and S/N in the thermody- 
namic limit can be also checked for network ensembles 
satisfying further constraints as for example the networks 
ensembles with given degree sequence and network com- 
munity structure, and network ensembles with given de- 
gree sequence and given spatial dependence of the net- 
works on the distance between the nodes. In general it 
is possible to demonstrate that as soon as we consider 
ensembles of networks with an extensive number of con- 
straints the Gibbs entropy per node E and the Shannon 
entropy per node S/N are non-equal in the thermody- 
namic limit. 

The von Neumann entropy of a network ensemble. In 
[l2| the authors have shown that is possible to define 



a von Neumann entropy of a network. This entropy is 
constructed from a density matrix p associated with the 
network. The density matrix must be a positive semi- 
definite matrix with unitary trace. In order to construct 
a density matrix from a network, in 12j it is proposed to 



consider the matrix p = L/J2ij a iji where L is the Lapla- 
cian matrix of the network, with Lij = J2 r a ir^i,j ~ a ij- 
The spectrum [l9| of the Laplacian matrix is important 
for the stability of 0(n) models, synchronization proper- 
ties of networks and determining the sca ling of the return 
times of random walk on the network [20(. Given p as 
specified above, we can calculate the average von Neu- 
mann entropy of an ensemble as 



Svn = -(Trplog(p))n ■ 



(12) 



The von Neumann entropy is therefore related to the 
spectra of the Laplacian. The theoretical evaluation of 
the self-averaging spectra of the Laplacian of complex 
networks ensemble is a very challenging topic that has 
attracted recent interest in the statistical mechanics com- 
munity [ig| | . Here we numerically explore how the von 
Neumann entropy Svn is related to the Shannon entropy 
of canonical ensembles. 

For G(N,p) networks the average von Neumann en- 
tropy, Sv n , is an increasing function of the average con- 
nectivity, pN, while the Shannon entropy per node, S/N, 
has the typical bell-shape form given by Eq. (JSJ), in 
the limit of large N. Therefore, for the G(N,p) ran- 
dom graphs ensembles, the relation between Svn and 
S is non monotonic, when we vary the average connec- 
tity p(N — 1). It is instructive to study the relation of 
the Shannon entropy of a network ensemble and its av- 
erage von Neumann entropy in networks with the same 
average degree. In 0] it has been shown that networks 
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FIG. 1: (Color online) The von Neumann entropy Svn versus 
the Shannon entropy per node S/N calculated for ensembles 
of scale-free networks with different expected average degree 
(k). The points are calculated by averaging over 20 networks 
in the ensemble of networks with N — 1000 nodes and differ- 
ent power-law exponents 7 of the distribution of the expected 
degrees -P(k) oc k '. The inset report the slope n defined in 
[13] as a function of (k) and the exponential fit indicated as 
a solid line. 



with power-law degree distribution P(k) oc fc -7 and con- 
stant average degree (k) have a Gibbs entropy per node 
E which is an increasing function of the power-law ex- 
ponent 7. Similarly the Shannon entropy per node S/N 
of canonical network ensembles with fixed expected de- 
gree Ki, where P(k) oc k -7 and fixed (k) is increasing 
with the power-law exponent 7. Therefore changing the 
power-law exponent 7 is a way to modulate S by leav- 
ing the average degree constant. In figure [T] we report 
the von Neumann entropy Svn vs. the Shannon entropy 
per node S/N in canonical power-law network ensembles 
with constant (re) and variable value of the 7 exponent. 
We find that the two entropies are linearly related 



S VN = r,S/N + p , 



(13) 



where 77 decays exponentially as a function of (k) for 
small values of (k) <C N. Therefore for scale free net- 
works the von Neumann entropy is linearly related to the 
Shannon entropy of the canonical ensembles measuring 
the number of typical networks in the ensemble. 

Conclusions In this Letter we have explored the con- 
nection between different definition of entropy of network 
ensembles. Interesting we have found that the Gibbs 
entropy per node E is equal to the Shannon entropy 
per node S/N in the thermodynamic limit for random 
graphs. However, when we consider networks with and 
extensive number of constraints (as for example a given 
degree distribution) the Gibbs entropy per node E and 
the Shannon entropy per node S/N differ by 0(1) terms. 
Moreover we have related the Shannon entropy with the 
recently introduced von Neumann entropy of networks. 



Interestingly we found that for scale free networks with 
constant average degree Svn an d S/N are linearly re- 
lated. We believe that all the entropies of the network 
ensembles, S and Svn as well as E 0] will play a cru- 
cial role for the quantification of the complexity and in 
inference problems in networks. 

G. B. acknowledge stimulating discussions with J. 
Baranyi, A.C.C. Coolen and A. N. Samukhin. 
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